After-action reviews for emergency preparedness and response to infectious disease outbreaks

After-action reviews (AARs) are a qualitative evaluation methodology conducted after a public health event recommended by the World Health Organization. Of 86 screened articles, we identified eight using AARs to evaluate public health responses to infectious disease outbreaks. Our findings suggested the toolkit is suitable to evaluate responses to infectious disease outbreaks but needs to be adapted for use in particular settings.

S ince early 2020, health systems around the world have faced challenges in adequately responding to the coronavirus disease (COVID- 19) pandemic. Systems have adapted to the evolving epidemic, and different measures have been implemented at different times in different contexts. Evaluating responses to significant public health events such as outbreaks of infectious diseases is often not prioritized or undertaken due to a lack of resources or time, despite its established importance in improving future preparedness and response measures. In 2015, the World Health Organization (WHO) developed the after-action review (AAR) toolkit as a component of the International Health Regulations (2005; IHR). [8][9][10][11] AARs aim to assess the what, how and why of a response to a significant public health event, to identify the best practices and challenges encountered during the response, and to propose mid-and long-term actions for improvement. The WHO AAR methods were developed to evaluate responses generally to any type of public health event. 11 AARs consist of nine pillars for which best practices, challenges and lessons learned are to be identified: (i) country-level coordination and monitoring, (ii) risk communication, (iii) surveillance, (iv) points of entry, (v) the national laboratory system, (vi) infection prevention and control, (vii) case management, (viii) operations and logistics and (ix) maintaining health services. 11 Conducting and reporting on an AAR requires three steps: (i) objective observation (i.e. a structured review of response activities); (ii) an analysis of gaps, best practices and contributing factors to the results of the response; and (iii) identification of areas for improvement and proposed follow-up actions. WHO suggests four methods that can be used to conduct an AAR: (i) debriefings, (ii) working groups, (iii) interviews with key informants and (iv) mixed-methods studies. Depending on the context, AARs can be conducted in different formats and cover different areas of the response. WHO also suggests that the findings of evaluations are compared against the IHR (2005) core capacities. 11 Final results should be summarized in a qualitative format, and evaluations by participants contributing to it are encouraged.
It is unclear to what extent WHO's AAR methods are being used to assess public health responses to events involving emerging infectious diseases and, in particular, how closely such evaluations follow WHO guidance. We undertook a rapid review of the global literature with the objective of understanding how the WHO AAR methods are being used to assess public health responses to infectious disease events.
Quach et al A rapid review of after-action reviews methodology Three AARs 13-15 used WHO-recommended methods in combination with other evaluation tools, such as document reviews or surveys in addition to quantitative assessments. The remaining five 16-20 strictly followed WHO's nine evaluation pillars and three steps, and were conducted as conferences that brought together all stakeholders. Four of the eight reports used working groups, 16-19 three used debriefings 13-15 and one used key-informant interviews, 20 following WHO's AAR readyto-use toolkits.
Public health systems were a common focus of evaluations, appearing in seven AARs, 13-19 while another AAR focused on a hospital setting. 20 Five AARs were conducted at the local level in response to outbreaks 13-15,19,20 and three at the national level. 16-18 Three AARs included participants' evaluations of and feedback on the AAR method. 17-19 Although the overall assessment of the suitability of AARs to connect stakeholders, provide a platform for ideas and to pool experiences was positive, as evidenced by responses from more than 80% of participants in each of these three studies, only half of the participants agreed that AARs actually achieved their objectives. [17][18][19] In terms of strengthening interdisciplinary collaboration and coordination, less than 20% of participants in these studies rated this as being accomplished by the AAR. [17][18][19] Additionally, the impor-the WHO Strategic Partnership for Health Security and Emergency Preparedness' After Action Review database, 12 WHO's main public repository for AARs. We included all articles and reports in English published or uploaded from January 2015 to December 2021 that described using WHO AAR methods to evaluate responses to infectious disease outbreaks. Reports or publications were excluded if they were incomplete, did not use the WHO AAR toolkit, were not published in English or did not evaluate infectious disease events. Results were merged, duplicates removed and the remaining reports screened against the inclusion and exclusion criteria, and reasons for exclusion were documented.
For the included reports, we extracted the key characteristics of the AAR method for use in a descriptive analysis. We also assessed how closely the included AARs followed the WHO AAR methods and how effective the methods were in assessing the response. The following data were extracted from each record: general information, including authors and year of publication; setting; scope of evaluation (national, regional or agency level); the event being evaluated; and the year of the event. The reports were then compared against WHO's AAR guideline ( Table 2). After screening 86 records, 8 were included in the analysis, 4 from the WHO AAR database and 4 peer-reviewed articles retrieved from PubMed ( Fig. 1, Table 3). From the completed reports, it is evident that WHO's methods were not always strictly followed, but they were often used in combination with other methods for qualitative and quantitative assessment. While the qualitative element of WHO's AAR toolkit seems to have been easier to follow in conference settings with all relevant stakeholders present, several AARs required methodological modifications, using, for example, surveys or document reviews, and also incorporated quantitative methods, depending on the local context. One of WHO's main recommendations for conducting AARs is to compare the outcomes of the response with the IHR (2005) core capacities -a country-level assessment -yet this comparison was not done in any of the studies included in our analysis.
WHO's AAR methodology is relatively broad and geared towards whole-of-system evaluations. The nine AAR evaluation pillars and the accompanying toolkit are also rather general. As a result, AARs were more frequently used for evaluations of district-and national-level systems rather than for specific systems (e.g. surveillance systems, national laboratories or point-of-entry screening). However, assessments should be conducted for all levels and aspects of health systems to ensure a comprehensive response; therefore, AAR toolkits should tance of allowing AAR methods to be adjusted to best fit their purposes (e.g. for smaller-level analyses, such as within a unit, region or single institution) is highlighted by the fact that three of the eight reports did not strictly follow WHO's AAR structure. 13-15 The importance of making modifications to conduct a more focused system evaluation was also flagged by Sorbello et al. 20 as a way to improve follow-up actions within local contexts and to enhance multidisciplinary cooperation. Despite being recommended by WHO, none of the AARs used the IHR (2005) core capacities as a comparator.
Only a limited number of AARs have been published in the scientific and grey literature. As of August 2022, the global public repository for AARs at WHO 12 listed 81 entries since 2016. However, 66 (88%) of the 75 entries categorized as having been conducted were incomplete, of which 41 were older than 2 years and hence are unlikely to ever be finalized. Furthermore, many entries had only minimal information about the setting and category of emergency, and were without much content about the AAR itself. It was also often difficult to establish whether an AAR had actually been conducted and completed successfully. The problem of identifying and accessing information about AARs has also been recognized in a recent review from Australia. 21  Quach et al A rapid review of after-action reviews methodology WHO issued a modified version of its methods for AARs at the beginning of the COVID-19 pandemic, known as intra-action reviews (IARs), to meet the need to rapidly assess health systems' performance during the ongoing pandemic. As of August 2022, there were 144 IARs listed in the WHO database; 22 129 of them (89.6%) were categorized as conducted, but only 19 of these (14.7%) were accompanied by a completed report, suggesting there are issues in finalization and publication similar to those for AARs. IARs include four additional pillars that are relevant to the ongoing COVID-19 pandemic: (i) COVID-19 vaccination, (ii) vulnerable and marginalized populations, (iii) national legislation and financing, and (iv) public health and social measures. 23 However, only two be flexible enough to be adapted to different jurisdictions and scopes of assessment to accommodate diverse evaluation needs. Thus, modifications to WHO's AAR guidance are important to ensure that relevant information can be gathered from a wider range of sources and a more diverse group of stakeholders to fully consider local contexts and different scopes of evaluation. Furthermore, understanding IHR (2005) core capacities could offer important lessons for conducting AARs. However, comparison against IHR (2005) core capacities is rarely done as part of AARs despite being encouraged by WHO. We also found that several AARs in the WHO repository were implemented without assessments from participants and stakeholders. of AARs, which hindered our ability to obtain sufficient information about their reflections on the suitability of the AAR method to achieve its objectives. All of these constraints could be resolved through more stringent reporting requirements for AARs. Unfortunately, there is no formal requirement to report on and publish AARs upon completion or to finalize reports in a timely manner. Especially during the COVID-19 pandemic, as more and more health systems need up-to-date data on effective and ineffective measures for addressing the pandemic, it is important to disseminate these evaluations widely and rapidly to ensure that incremental and strategic improvements are made to health-care systems worldwide. However, it seems that COVD-19-specific IARs suffer from the same issues as AARs in terms of insufficient conclusions and lack of publication of reviews.
It is crucial to evaluate public health systems regularly during a prolonged and evolving event such as the COVID-19 pandemic. Selecting appropriate methods for these evaluations is important to their successful implementation and to ultimately improve and adapt responses to the pandemic. Considering the variability of the COVID-19 pandemic and countries' public health capacities, a global methodology such as WHO's AAR toolkit needs to be sufficiently adaptable to local contexts and priorities, and also able to gain the most value from stakeholders' practical experiences during the response. The COVID-19-specific IAR adaptation of the AAR is a laudable example of this type of approach, and future pandemics might require similar adaptations. Furthermore, more subnational reviews, which have been proposed in the latest version of the IAR, are needed to enable better operational analysis of public health responses in specific high-priority areas. Importantly, the reporting and publication of completed AARs should be strengthened to allow public health responders and researchers from other countries and settings to benefit from the knowledge generated and lessons learned to strengthen the capacities of health-care systems to respond to future health emergencies. 23

Conflicts of interest
The authors have no conflicts of interest to declare.

Ethics statement
Not applicable.
IARs categorized as conducted in the database included information about COVID-19 vaccination, and none of the IARs provided information about the other three pillars. Therefore, it is unclear whether IARs have contributed to improving evaluations of health system responses. The IAR adaptation of the AAR remains relatively broad and geared towards national-level responses. Similar to AARs, we believe that IARs would greatly benefit from regular evaluation of the methodology itself to better guide and prepare countries and health-care systems for future, protracted health emergencies beyond the COVID-19 pandemic. 23 We acknowledge several limitations to our work. First, the small number of included records did not allow for strong conclusions. Second, there were many AARs listed in the WHO repository that did not have a completed report, which again led to only a small number of records being included in our study and possible publication bias in our assessment. Third, the range of countries with completed AARs was limited and quite focused on WHO's African Region, which restricts the generalizability of our findings. Fourth, less than half of the included studies reported on participants' evaluations due to wrong focus (not infectious diseases) due to wrong study design not in English incomplete not conducted